Skip to content

Record whether a command's last run failed (fixes #187) - #197

Open
typeless wants to merge 2 commits into
mainfrom
fix/187-remember-failed-commands
Open

Record whether a command's last run failed (fixes #187)#197
typeless wants to merge 2 commits into
mainfrom
fix/187-remember-failed-commands

Conversation

@typeless

Copy link
Copy Markdown
Owner

A command that exited nonzero was not remembered. If its declared outputs existed when the next build started, that build reported "Nothing to do (up to date)" and exited 0 — the failure simply disappeared:

build 2 (-k):  Build completed: 0 commands (1 failed)   exit 1
build 3 (-k):  Nothing to do (up to date).              exit 0

Why it was wrong, from the bisect

The bisect on the issue localises this to 6ae36d1c2, which made the index write unconditional and justified it in the commit message:

Failed outputs don't exist on disk, so stat fails and they're detected as changed.

True only when the failed command produced nothing. cp %f %o && false writes its output and then fails, so the stat succeeds, the hash matches what the index just recorded, and there is nothing left to notice.

The guard it replaced was stats.failed_jobs == 0 && !opts.dry_run — exit status was honoured, bluntly, by not writing the index at all. That commit removed it while arguing about a different category of recorded state. This is the shape #189 describes: with the categories scattered, a plausible local argument can remove a leg rather than merely forget one.

The fix

Exit status becomes what it should have been — a recorded field. CommandEntry carries failed, set from the scheduler's job results, and a command whose previous entry is marked failed re-runs regardless of its signature.

Restoring the old guard was the alternative and is worse: it also discards the successful commands' state, which is the thing 6ae36d1c2 legitimately fixed.

INDEX_VERSION 19 → 20; RawCommandEntry gains a flags word, 80 → 88 bytes.

Routing is not interchangeable here

The re-run is pushed through changed_outputs, not forced_cmds. forced_cmds are OR'd into the affected set at cmd_build.cpp:2058 after collect_affected_commands has computed its closure over changed files, so a forced id adds exactly one command and its consumers do not propagate. A failed command that half-wrote its output has to reschedule whatever read that output.

This was flagged on #189 as one of two constraints the bisect explicitly did not measure, so it is verified here by test rather than assumed.

Tests

Two e2e scenarios, both observed failing first. The second pins the routing: a producer writes PARTIAL and fails, its consumer copies that into the final artifact, and once the cause of the failure is removed — with nothing else changing — the producer must re-run and the consumer must pick up the corrected output.

Before this change that assertion read:

final.txt: "PARTIAL"    with the build reporting success

Also verified by hand: the failure survives repeated builds (3 and 4 both exit 1), and clears on the first successful run, after which the next build is a no-op — so this does not trade a stuck-green for a stuck-red.

Verification

make test: 142417 assertions, 639 cases, 32 e2e shards. make tidy, make iwyu clean.

Noted, not fixed

The new verbose line reads Failed last run: with an empty name, because Display is only set from ^ ^ markers. That affects every -v routing diagnostic, not just this one, and is worth its own change.

🤖 Generated with Claude Code

https://claude.ai/code/session_01TLob4Ef3qyMd3Dnp9q3XmA

A command that exited nonzero was not remembered. If its declared outputs existed
when the next build started, that build reported "Nothing to do (up to date)" and
exited 0, so the failure disappeared:

  build 2 (-k):  Build completed: 0 commands (1 failed)   exit 1
  build 3 (-k):  Nothing to do (up to date).              exit 0

The bisect on the issue localises it to 6ae36d1, which made the index write
unconditional and justified it in the commit message:

  Failed outputs don't exist on disk, so stat fails and they're detected as changed.

That holds only when the failed command produced nothing. `cp %f %o && false`
writes its output and then fails, so the stat succeeds, the hash matches what the
index just recorded, and there is nothing left to notice. Before that commit the
guard was `stats.failed_jobs == 0 && !opts.dry_run` -- exit status was honoured
bluntly, by not writing the index at all, and that commit removed the guard while
arguing about a different category.

So exit status becomes what it should have been: a recorded field. CommandEntry
carries `failed`, set from the scheduler's job results, and a command whose
previous entry is marked failed re-runs regardless of its signature. Restoring the
old guard was the alternative and is worse -- it also discards the successful
commands' state, which is the thing 6ae36d1 legitimately fixed.

Routing matters here and is not interchangeable. The re-run is pushed through
`changed_outputs`, not `forced_cmds`: forced_cmds are OR'd into the affected set
at cmd_build.cpp:2058 *after* collect_affected_commands has computed its closure
over changed files, so a forced id adds exactly one command and its consumers do
not propagate. A failed command that half-wrote its output must reschedule
whatever read that output.

INDEX_VERSION 19 -> 20: RawCommandEntry carries a flags word, 80 -> 88 bytes.

Two e2e scenarios, both observed failing first. The second is the one that pins the
routing: a producer writes PARTIAL and fails, its consumer copies that to the final
artifact, and after the cause of the failure is removed -- with nothing else
changing -- the producer must re-run and the consumer must pick up the corrected
output. Before this change that assertion read "PARTIAL", with the build reporting
success.

Also verified by hand: the failure survives repeated builds (3 and 4 both exit 1),
and clears on the first successful run, after which the next build is a no-op.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TLob4Ef3qyMd3Dnp9q3XmA
@github-actions

github-actions Bot commented Jul 28, 2026

Copy link
Copy Markdown

PR metrics

Performance (gcc example, Linux)

Workload Instructions CPU time Page faults D1 miss LL miss Wall Peak RSS
parse 1076 M 0.28 s 14.2 k 0.6% 0% 0.287 s 31.2 MB (-0.1MB)
dry-run 1881 M 0.36 s 16.6 k 0.7% 0% 0.374 s 40.2 MB

Deterministic signals: instructions (cachegrind-simulated instruction reads — exact across runs, no PMU needed), page faults, peak RSS, and the cachegrind D1/LL miss rates. CPU time is user+sys from time(1).

Internal statistics (gcc example, up-to-date dry run)

Metric Value
Tupfiles parsed 24
Commands 3545
Commands scheduled 0
Files checked 5818
Files changed 0
Files in index 6087
Graph edges 367913
Index size (bytes) 7406610 (+0.4%)
Implicit deps 330624
Hash computations 0
Hashes skipped (stat cache) 5818
Stat calls 5869
Parse time (ms) 282.4 (+19.7%)
Total time (ms) 363.5 (+18.6%)
Runner CPU INTEL(R) XEON(R) PLATINUM 8573C

Counters from putup -n --stat on the fully-built gcc example (up-to-date dry run): deterministic work measures — a jump in commands scheduled, hash computations, or stat calls is a real behavior change, not noise. Timings are the minimum over repeated runs, compared only against a baseline from the same CPU model; the counters are the regression signal.

Binary size (Linux)

Binary .text .data .bss File
putup 518.6 KB (+0.2%) 1.6 KB 98.7 KB 615.4 KB

Code churn (whole codebase, last 30d)

Files Lines written Still present Churned Churn rate
119 7554 6768 786 10.4%

Of the lines written across the codebase in the last 30 days, how many are already gone — work that was written and then discarded or rewritten inside the same window. This is the state of the tree including this PR, not a measure of the PR itself. Only code we write is counted: tests, examples, vendored and generated files, CI plumbing and prose are excluded. 3585 lines were deleted in the window in total, most of them older than it.

Where the churn is
File Lines written then discarded
src/cli/cmd_build.cpp 255
src/core/layout.cpp 69
src/core/new_delete.cpp 63
src/cli/context.cpp 55
src/graph/builder.cpp 44
src/core/string_pool.cpp 34
src/core/heap_buf.cpp 33
src/platform/vm-posix.cpp 22
src/cli/options.cpp 21
include/pup/core/layout.hpp 15

Test coverage (lines)

Overall Median file Min file Max file
86.7% 95% 0.0% src/platform/path-posix.cpp 100.0% include/pup/core/arena.hpp

103 files · 15185/17508 lines covered

Deltas vs main@a1277e1d4.

Updated for 7050628

From an adversarial review of this PR. The field said "failed during the run that
wrote this index" when it must say "has not succeeded since it last failed", and
that difference reopened #187 through two doors, both reproduced:

  putup -B . -k             # a/ fails, recorded
  putup -B . -k outb.txt    # target build; a/ does not run
  putup -B . -k             # Nothing to do (up to date).   exit 0

and the same via a scoped build, where merge_out_of_scope_commands copied the old
entry forward without its failed flag. tup remembers in both cases; this forgot.

The flag is now seeded from the previous index at detection, carried through any
build in which the command does not run, and cleared only by a successful run of
that command. merge_out_of_scope_commands carries it too, unconditionally: an
out-of-scope command did not run, so its recorded state cannot have advanced.

The consumer scenario did not pin what the commit message claimed it pinned.
Routing retries through forced_cmds -- the alternative the message argues against
-- left both scenarios passing, because final.txt had never been created and the
consumer was scheduled by missing-output detection whatever the routing did. It now
builds successfully first so the consumer's output exists holding V1, and asserts
it reaches V2. Verified by mutation: that routing change now fails the test.

Two scenarios added for the doors above, and the flags/reserved comments corrected
-- there is no CommandFlags type, and Hash256 is std::array<std::byte, 32> so
nothing needed 8-byte alignment; reserved makes the tail padding explicit.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TLob4Ef3qyMd3Dnp9q3XmA
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant